The present disclosure relates to an image processing device, an image processing method, and a program. In particular, the present disclosure relates to an image processing device, an image processing method, and a program that generate an image with a dynamic range that is wide.
A solid image capture element that is used in a video camera or a digital still camera, such as a CCD image sensor or a complementary metal oxide semiconductor (CMOS) image sensor, accumulates an electrical charge that corresponds to the amount of incident light and performs a photoelectric conversion that outputs an electrical signal that corresponds to the accumulated electrical charge. However, there is an upper limit to the amount of the electrical charge that is accumulated in the photoelectric conversion element, and when more than a fixed amount of light is received, the amount of the accumulated electrical charge reaches a saturation level, such that what are called blown out highlights, which are set at a saturated brightness level, occur in regions of the photographic subject where the brightness is greater than a fixed value.
In order to prevent this sort of phenomenon from occurring, processing is performed that adjusts the exposure time by controlling the period during which the electrical charge is accumulated in the photoelectric conversion element, in accordance with variations in the outside light and the like, and that also adjusts the sensitivity to an optimum value. For example, for a bright subject, the exposure time is shortened by using a faster shutter speed, shortening the period during which the electrical charge is accumulated in the photoelectric conversion element, and the electrical signal is output before the amount of the accumulated electrical charge reaches the saturation level. This sort of processing makes it possible to output an image that accurately reproduces the gray-scale levels of the subject.
However, in capturing an image of a subject that has a mixture of bright regions and dark regions, using a fast shutter speed means that the exposure time will not be sufficient for the dark portions, so the signal-to-noise ratio worsens, and the image quality deteriorates. In order to accurately reproduce the brightness levels of the bright portions and the dark portions in a captured image of a subject that has a mixture of bright regions and dark regions, processing must be performed that increases the exposure time and achieves a high signal-to-noise ratio for the image sensor pixels where the amount of the incident light is low and that avoids saturation in the pixels where the amount of the incident light is high.
As a technique for implementing this sort of processing, a technique is known that sequentially captures and combines a plurality of images with different exposure times. Specifically, an image with a long exposure time and an image with a short exposure time are captured separately in sequence. The technique generates a single image by performing combining processing that uses the long exposure time image for the dark image regions and uses the short exposure time image for the bright image regions where the highlights are blown out in the long exposure time image. Combining a plurality of images with different exposures in this manner makes it possible produce an image with a dynamic range that is wide and in which there are no blown out highlights, that is, a wide dynamic range image (a high dynamic range (HDR) image).
For example, Japanese Patent Application Publication No. JP-A 2008-99158 discloses a configuration that produces a wide dynamic range image by combining a plurality of images with different amounts of exposure. The processing will be explained with reference to FIG. 1. An image capture element, in capturing moving images, for example, outputs image data for two different exposure times within a video rate (30 to 60 fps). In capturing still images, too, the image data are generated for two different exposure times and output. FIG. 1 is a figure that explains characteristics of images (a long exposure time image, a short exposure time image) that the image capture element generates and that have two different exposure times. The horizontal axis is time (t), and the vertical axis is an accumulated electrical charge (e) in a light-receiving photo diode (PD) that configures a photoelectric conversion element that corresponds to one pixel of a solid image capture element.
For example, in a case where the amount of light that the light-receiving photo diode (PD) is large, that is, where it corresponds to a bright subject, the accumulated electrical charge increases rapidly as time elapses, as shown in a high brightness region 11 that is shown in FIG. 1. In contrast, in a case where the amount of light that the light-receiving photo diode (PD) is small, that is, where it corresponds to a dark subject, the accumulated electrical charge increases slowly as time elapses, as shown in a low brightness region 12 that is shown in FIG. 1.
The time from t0 to t3 is equivalent to an exposure time TL for acquiring the long exposure time image. The line that is shown in the low brightness region 12 shows that the accumulated electrical charge at the time t3, even as the long exposure time TL, has not reached the saturation level (unsaturated point Py), and an accurate gray-scale expression can be produced according to the gray level of the pixel that is set using an electrical signal that is produced based on the accumulated electrical charge (Sa).
However, the line that is shown in the high brightness region 11 clearly indicates that the accumulated electrical charge has already reached the saturation level (saturated point Px) before it reaches the time t3. Therefore, in the high brightness region 11, only a pixel value that corresponds to an electrical signal at the saturation level is produced from the long exposure time image, resulting in a pixel that is blown out.
Accordingly, in the high brightness region 11, the accumulated electrical charge is swept out of the light-receiving photo diode (PD) once before the time t3 is reached, for example, at a time t1 (a charge sweeping starting point P1) that is shown in FIG. 1. The charge sweeping does not sweep out the entire accumulated electrical charge in the light-receiving photo diode (PD), but sweeps it down to an intermediate voltage hold level that is controlled by the photo diode (PD). After the charge sweeping processing, the light-receiving photo diode (PD) is once again exposed to light for a short time that is defined as an exposure time TS (from t2 to t3). That is, a short time exposure is made for the period from a short exposure time starting point P2 to a short exposure time ending point P3, which are both shown in FIG. 1. An accumulated electrical charge (Sb) is produced by the short time exposure, and the gray level of the pixel is set using an electrical signal that is produced based on the accumulated electrical charge (Sb).
Note that in the setting of the pixel value using the electrical signal that is based on the accumulated electrical charge (Sa) that is produced by the long time exposure in the low brightness region 12 and using the electrical signal that is based on the accumulated electrical charge (Sb) that is produced by the short time exposure in the high brightness region 11, an estimated accumulated electrical charge is computed for a case in which the exposure times are the same in the two regions, an electrical signal output value that corresponds to the estimated accumulated electrical charge is computed, and the pixel value is set based on the results of the computations.
Combining the short exposure time image and the long exposure time image in this manner makes it possible to produce an image that has no blown out highlights and a dynamic range that is wide.
Furthermore, Japanese Patent Application Publication No. JP-A 2000-50151 discloses a configuration that captures a plurality of images with different amounts of exposure, in the same manner as described in Japanese Patent Application Publication No. JP-A 2008-99158, and in the performing of the combining processing, inhibits the occurrence of false color that is associated with the combining by comparing the plurality of the images with the different amounts of exposure, specifying a pixel region where there is movement, and performing correction.
However, with the configurations that are described in Japanese Patent Application Publication No. JP-A 2008-99158 and Japanese Patent Application Publication No. JP-A 2000-50151, it is necessary to capture the long exposure time image and the short exposure time image individually and perform the processing that combines them.
In this manner, using a plurality of images for which the exposure times have been changed makes it possible to produce the wide dynamic range image (the high dynamic range (HDR) image), but the problems that are described below, for example, occur in the processing that is based on the plurality of the images.
First problem: The image capture must be performed a plurality of times.
Second problem: A plurality of images that have been captured at different times are combined, and captured image data with long exposure times are used, so the process is vulnerable to the effects of camera instability.
Known technologies for solving these problems include the known technologies described below, for example.
Technique for Solving First Problem
The pixels within a single solid image capture element are set in two types of exposure patterns, that is, pixels with two different types of exposure time control, pixels that are exposed for a short time, and pixels that are exposed for a long time. This processing makes it possible to capture images with the short time exposure and the long time exposure pixels almost simultaneously.
For example, this sort of configuration is disclosed in Japanese Patent Application Publication No. JP-A 2006-311240 and Japanese Patent Application Publication No. JP-A 2006-253876.
However, with these configurations, there is a problem in that the exposure times vary from one pixel to the next, so the long exposure time pixels are more susceptible to blurring than the short exposure time pixels, and it is difficult to completely eliminate the effects of camera instability.
Technique for Solving Second Problem
For example, “SIGGRAPH 2007: Image Deblurring with Blurred/Noisy Image Pairs” proposes a technique for producing a high-quality image from a short exposure time image with a lot of noise and a long exposure time image that has blurring. However, even with the disclosed technique, it is necessary to perform image capture twice, once with a short exposure time and once with a long exposure time.
Japanese Patent Application Publication No. JP-A 2010-109948 proposes a technique that, by estimating an amount of movement based on a plurality of images, corrects blurring in a captured image that has been captured with a wide dynamic range.
Furthermore, in “Coded Rolling Shutter Photography: Flexible Space-Time Sampling (ICCP2010),” a technique is disclosed that simultaneously performs image stabilization and processing that combines wide dynamic range images (HDR combining), based on image data that have been captured by varying the exposure times within a single solid image capture element one line at a time.
However, in order for the amount of blurring to be estimated when the exposure times are varied one line at a time, the image must be captured by combining pixels with at least three different exposure times, giving rise to a problem of blurring in the vertical direction. Moreover, the technique here described involves processing of an image after demosaicing, in which the RGB values have been set for each of the pixel positions, so a color array such as a Bayer array or the like is not taken into consideration.
Japanese Patent Application Publication No. JP-A 2010-62785 proposes a technique that, by implementing a control method for the solid image capture element, intermittently captures images with short exposure times while a long exposure time image is being captured. However, there is a problem in that the short exposure time image capture must be performed a plurality of times, which makes the control more complicated.